LEGAL ISSUES AND RISK MANAGEMENT 


When the Creeks Rise: Disaster 
Recovery in Cedar Rapids 

What do you do when your school buildings are flooded and your 
systems are in jeopardy ? You act fast and learn from your mistakes. 

By Dave Brousard 



T hursday, June 12, 2008, dawned as another 

beautiful morning in Hawaii. There was a slight 
breeze, and the projected high was in the mid- 
80s with a chance of a shower in the afternoon. 
I couldn’t wait to get moving. 

Then the phone rang and everything changed. 

It was my brother in Cedar Rapids, Iowa. He had just 
waded a half mile through chest-high water carrying 
only his cat and had taken shelter at the middle school. 

I couldn’t really comprehend what he was telling me. 
Water that high? At his house? It just didn’t make sense. 


He lives 10 blocks from the river and had never had 
water in his basement. But now it was up to his win¬ 
dowsill? He told me to turn on CNN and sure enough, 
there was my hometown. 

My family and I sat glued to the TV for a good part 
of the day. Every camera shot seemed to be of some 
place we knew. One-fifth of the city of Cedar Rapids 
was under water. Thankfully, there were no fatalities. 

At about 7:00 p.m., I decided to check my work 
email—I am the director of information technology at 
Grant Wood Area Education Agency in Cedar Rapids— 
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Left: Dryers helped dry the previously flooded rooms. 
Right: Servers were moved to a temporary location so 
systems could be brought back online. 


but I couldn’t connect. I thought that was odd so I tried 
calling up my Grant Wood home page. Still no response. 

Back home in Iowa it was 1:00 a.m., so 1 couldn’t call 
anyone to find out what was happening. I did try calling 
my work number to see if someone had left me a mes¬ 
sage. This time, I got a fast busy signal. I didn’t have a 
good feeling. 

I stayed up that night and watched the news until 
2:00 a.m.—which was 8:00 a.m. back home. Then I 
called Al, our network engineer and one of the most 
dedicated and knowledgeable people I know. He said it 
was bad, and there might be three feet of water in the 
building. The local creek had flooded the building 
because it had nowhere to flow because the river was so 
high. The good news, he told me, was that he and two 
other people moved the four racks of computers to the 
second floor. The bad news was some of them had not 
shut down gracefully, so there was no telling how they 
might restart. We didn’t know when we might be able 
to get in the building. 

Saturday, June 14, 2008 

We flew into Cedar Rapids on Saturday morning. The 
view from the air was incredible. Water was everywhere. 
We landed, got our bags, and went straight home. I 
dropped off my family and headed to work. I had no idea 
what I would see or if I could even get in the building. 

As I approached, I saw that the water had retreated 
enough so I could get to the building. Pulling up to the 
entrance to my department, I could see that the water in 
the building had been a foot deep. I opened the door and 
peered in. The electricity was still on, so I turned on the 
lights and went in. I was the first one there. 

Mud covered the floor and stuff had floated every¬ 
where. I detected a slight smell of smoke and went 
farther into the building. Peering through the window 
of one of the hallway doors, I saw that smoke filled the 


entire hallway. I called the fire department; within five 
minutes, an entire assembly of fire trucks had arrived. 

I led the firefighters around, and we finally found a 
surge protector that had been underwater and had 
shorted out. They deemed the building safe and left. 

I could see that the water in the 
building had been a foot deep. 

My attention now turned toward those servers on the 
second floor. How could we get them functional so we 
could start doing payroll for the 60 school districts that 
we service? Those districts understood, but that under¬ 
standing wouldn’t last long if they missed a payroll and 
staff members couldn’t pay their bills. 

My first phone call was to Al to get his thoughts. He 
suggested that we move the servers back to the first floor 
where there was appropriate power and Ethernet. Then 
we could start turning some of them on. 

The elevator was on the first floor, the doors were open, 
and it had power. However, the floor outside the elevator 
was covered in some type of oil. Not knowing anything 
about elevators, 1 called our building engineer. He directed 
me to the maintenance closet on the ground floor, which 
housed the elevator pump. The elevator pump had been 
submerged in water and was now coated with the same 
oily substance. I was in way over my head, so I called the 
elevator service department. They asked if I was willing to 
pay for an emergency call. I had no choice. 

When the service person arrived less than an hour 
later, I took him to the closet; he shook his head. The 
oil was hydraulic fluid. The motor needed to be removed 
and rebuilt. Next, he looked at the elevator and in¬ 
formed me that we should have left it on the second 
floor so critical components wouldn’t have gotten wet. 
He’d be back the next day (Sunday), but in the mean- 
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time, I needed to clean up the fluid and disconnect the 
motor so he could take it to be rebuilt. I also needed to 
pump the water out of the elevator shaft. 

Off to work I went. For the expense of a Wet/Dry Vac, 
I removed all the fluid, cleaned out the maintenance 
closet, and disconnected the motor. About that time, A1 
appeared, as did another employee, and we began clean¬ 
ing up our server room—which was hard to do, given 
the city ban on clean water use. However, we got the job 
done and finally called it a night at 11:00 p.m. It had 
been a long day. 

Sunday, June 15, 2008 

I was up and at the home improvement store by 9:00 
a.m. to buy a pump. I spent Sunday cleaning out the ele¬ 
vator shaft, helping remove the motor, and recleaning 
our computer room where water had seeped back in. 

We went home at 11:00 p.m. 

Monday June 16, 2008 

Monday was the day we hoped to have our elevator 
working so we could move those four racks of servers 
back to their home. However, the elevator company said 
that wouldn’t happen until Tuesday. Since there was 
nothing we could do, we set about helping others who 
had arrived clean up their areas. 

At about noon, Al had an idea. We could get our 
email and our Web server going and that would be our 
first little victory. We grabbed some extension cords and 
some Ethernet cables, and several hours later we had 
email. That one little success really boosted morale. 

We were connected to the world again! 

Tuesday June 17, 2008 

Tuesday was our day. The elevator motor was installed, 
and the elevator repairman was able to get all our equip¬ 
ment back to the first floor. By noon, we were plugging 
in servers and by 7:00 p.m., we had everything back up 
and running. The building was still a mess and would 
be for the next year. We moved our servers three more 
times, but those moves paled compared with the uncer¬ 
tainty of those first few days. 

What Did We Learn? 

You need capable, dedicated staff who are not afraid to 
make decisions without a road map. Right or wrong, 
they must feel comfortable doing so, knowing that you 
will support them no matter what happens. I had that 
person in Al. Instead of calling it quits and going home, 
he made some tough decisions, which in the end were 
the right decisions. 

We have made a lot of changes as a result of what we 
learned that day in 2008. The most important is the 


12-by-27-foot telecommunications shelter we purchased 
to house our servers. This tornado-proof shelter is raised 
four feet above the level of the 2008 flood and has 
redundant air-conditioning. Recently, another area edu¬ 
cation agency (AEA) purchased the same type of shelter. 
We each have the same kind of facility and the same 
backup system; so each evening, we back up our data 
not only to the backup system at our location but also 
to the identical backup system at the sister agency. 

You need capable, dedicated 
staff who are not afraid to make 
decisions without a road map. 

Our agreement with that other agency also states that 
each will provide a server that can be virtually subdi¬ 
vided to run email, Webmail, DNS, and our finance and 
payroll systems. So should such a catastrophe happen 
again at either agency, we now have a location where we 
can begin recovery and restore our hardware and data. 

These critical pieces should be up and functioning 
within four hours. That time frame includes the two- 
hour drive to the other agency and also assumes that 
the staff there will get started as soon as they get the 
call for help. 

This process is tested twice a year at each location. 
After the test, staff members from each agency sit down 
and discuss what worked and what didn’t and ways we 
can make the entire process more effective. 

One of our remaining vulnerabilities is having more 
than 60 servers at our location. We cannot feasibly have 
that many servers at our sister agency. So we are explor¬ 
ing the possibility of developing some type of disaster 
recovery consortium that any Iowa AEA or local educa¬ 
tion agency (LEA) could join. The only stipulation to 
membership would be for each organization to make 
one server available at its location that meets a set of 
predetermined criteria that could be rededicated for dis¬ 
aster recovery for some other member of the group if 
necessary. This type of consortium would then provide 
more servers that could be used should a disaster happen 
to a member of the AEA or LEA. 

The next time you drive by your nearest body of 
water, whether it is a creek, lake, river, or ocean, try to 
find its high-water mark. Now imagine what would hap¬ 
pen if the water exceeded that spot by 10 feet. That’s 
what happened in Cedar Rapids, Iowa, in June 2008. 

Are you prepared? 


Dave Brousard is director of information technology for Grant 
Wood Area Education Agency in Cedar Rapids, Iowa. Email: 
dbrousard@gwaea.org 
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